HAWQ-1658. Easy Hawq and PXF initialization #1396

eschizoid · 2018-09-02T13:46:58Z

Hey Hawq team, thanks for this great project and congrats on becoming TLP 😄

Over the last couple of days I've been using Hawq. However, the information on how to use/install it's not yet centralized on a single place (some bits can be found on confluence, some parts on the Pivotal official Hawq guide, and finally some bits on the Apache official Hawk site).

The purpose of this PR is to provide build init start stop status commands for the following services:

Hawq
PXF

Please let me know what you think, and if you will be interested in this. Right now I only added support for centos7 docker container, but I can quickly add the same support for centos6 images.

radarwave · 2018-09-05T03:44:49Z

Please start a jira first, so we can discuss if all the new features are reasonable. Maybe we can also discuss if anything we can leverage of the current docker environment, e.g. run feature tests.
Please start the PR after local tests passed. Thanks.

eschizoid · 2018-09-05T19:30:47Z

@radarwave I do believe make build runs all the core tests, and make pxf runs all the gradle tests for PXF. Is there any other set of tests that I could be missing?

All the above tests are passing as far as I can tell.

radarwave · 2018-09-06T09:52:58Z

@eschizoid I asked more tests because commands in start-hawq.sh would not work. Now seems they are not used by any make commands.

contrib/hawq-docker/centos7-docker/hawq-test/start-hawq.sh

eschizoid · 2018-09-07T15:24:13Z

Hey @radarwave I think all the PR comments were addressed and I added tests for starting Hawq and PXF as you requested. Please let me know what you think.

ginobiliwang · 2018-09-10T03:00:50Z

@eschizoid the "start-hawq.sh" script is a little bit confused, since it is more like to initiate and bootstrap
a hawq cluster.

Meanwhile, we may need to treat start/stop cluster which are regular maintenance actions and initiation of a cluster as two different scenarios.

contrib/hawq-docker/centos7-docker/hawq-test/make-hawq.sh

contrib/hawq-docker/Makefile

eschizoid · 2018-09-10T13:54:13Z

@ginobiliwang

the "start-hawq.sh" script is a little bit confused, since it is more like to initiate and bootstrap
a hawq cluster.

Agreed, I will make two separate bash scripts, one for initializing and one for starting PXF and Hawq, respectively

Meanwhile, we may need to treat start/stop cluster which are regular maintenance actions and initiation of a cluster as two different scenarios.

Agreed

eschizoid · 2018-09-17T16:44:21Z

@ginobiliwang @radarwave Let me know what you think about this new approach and if this satisfies all the requirements. Sorry for the wait.

radarwave · 2018-09-26T09:16:23Z

Need more error message to guide user what the correct steps to begin, 'make init-hawq' is good example, we should add error messages to other make options too. See:
`make init-hawq
Make sure you have executed make hawq
Logging into centos7-namenode container
centos7-namenode container does not exist!

make hawq
Logging into centos7-namenode container
centos7-namenode container does not exist!

make start-hawq
Logging into centos7-namenode container
centos7-namenode container does not exist!
`
Per my check, I need to run as below sequences:
0. make build (Optional, this to make sure the docker image in local is build by the latest code.)

make run
make hawq
...

contrib/hawq-docker/Makefile

contrib/hawq-docker/centos7-docker/hawq-test/service-hawq.sh

radarwave · 2018-09-26T09:49:18Z

Anyone can help to review on pxf side?

leskin-in · 2018-09-26T12:05:06Z

contrib/hawq-docker/centos7-docker/hawq-test/service-pxf.sh

+EOF
+
+ rm -rf ${PXF_HOME}/conf/pxf-private.classpath
+ cat <<EOF >>${PXF_HOME}/conf/pxf-private.classpath


If https://github.com/apache/hawq/blob/master/pxf/pxf-service/src/configs/templates/pxf-private-hdp.classpath.template changes, someone who does that may forget to make the same changes in the current script.

I wonder whether it is really necessary to replace ${PXF_HOME}/conf/pxf-private.classpath with the configuration below. If so, I would propose to generate classpath from the template (right here).

@leskin-in I think we can use some inspiration from #1335. Have a make variable with the Hadoop distro (cdh, hdp, emr, etc) and load the template during build-time. That way we only have one centralized place with all the classpath templates, and each process can load them however they want. Let me know what you think.

I wonder whether it is really necessary to replace ${PXF_HOME}/conf/pxf-private.classpath with the configuration below.

Unfortunately, this is needed :(

Good, I think this code does the classpath generation mentioned. 👍 for your solution.

leskin-in

IMHO, PXF part looks fine. Thanks @eschizoid !

eschizoid · 2018-09-27T14:20:11Z

@radarwave I Added better error messages for all the new make operations. Let me know what you think. Hopefully we can merge this PR soon 😄.

eschizoid · 2018-09-27T14:27:56Z

contrib/hawq-docker/Makefile

@@ -51,7 +65,10 @@ build-hawq-dev-$(OS_VERSION): $(TOP_DIR)/$(OS_VERSION)-docker/hawq-dev/Dockerfil

 build-hawq-test-$(OS_VERSION): $(TOP_DIR)/$(OS_VERSION)-docker/hawq-test/Dockerfile
 @echo build hawq-test:$(OS_VERSION) image
- docker build -t hawq/hawq-test:$(OS_VERSION) $(TOP_DIR)/$(OS_VERSION)-docker/hawq-test/
+ docker build \
+ --build-arg=PXF_CLASSPATH_TEMPLATE="`cat ../../pxf/pxf-service/src/configs/templates/pxf-private-${PXF_CLASSPATH_TEMPLATE}.classpath.template`" \


This might look like a hack, but I found it handy for loading files from a different docker build context. Happy to change it if there is a better way of doing this.

radarwave · 2018-09-28T05:36:17Z

Much better. +1

Thanks for making build/run hawq easier.

eschizoid · 2018-09-28T09:51:27Z

@radarwave You think we can merge down this PR?

radarwave · 2018-09-28T09:55:24Z

Sure, merged. Please close this PR.

radarwave reviewed Sep 6, 2018

View reviewed changes

eschizoid force-pushed the development branch from 7462e3b to fda0229 Compare September 7, 2018 19:14

radarwave reviewed Sep 10, 2018

View reviewed changes

eschizoid force-pushed the development branch 2 times, most recently from c3ae8e2 to d10c5c1 Compare September 17, 2018 16:41

eschizoid changed the title ~~Easy Hawq and PXF initialization~~ HAWQ-1658 - Easy Hawq and PXF initialization Sep 21, 2018

eschizoid changed the title ~~HAWQ-1658 - Easy Hawq and PXF initialization~~ HAWQ-1658. Easy Hawq and PXF initialization Sep 21, 2018

radarwave reviewed Sep 26, 2018

View reviewed changes

contrib/hawq-docker/Makefile Outdated Show resolved Hide resolved

contrib/hawq-docker/centos7-docker/hawq-test/service-hawq.sh Outdated Show resolved Hide resolved

leskin-in reviewed Sep 26, 2018

View reviewed changes

eschizoid commented Sep 27, 2018

View reviewed changes

eschizoid force-pushed the development branch from 52b5f9f to e04e53c Compare September 27, 2018 15:55

Added support for HAWQ and PXF installation

5472ff5

eschizoid force-pushed the development branch from e04e53c to 5472ff5 Compare September 27, 2018 20:44

eschizoid closed this Sep 28, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HAWQ-1658. Easy Hawq and PXF initialization #1396

HAWQ-1658. Easy Hawq and PXF initialization #1396

eschizoid commented Sep 2, 2018 •

edited

Loading

radarwave commented Sep 5, 2018

eschizoid commented Sep 5, 2018 •

edited

Loading

radarwave commented Sep 6, 2018

eschizoid commented Sep 7, 2018

ginobiliwang commented Sep 10, 2018

eschizoid commented Sep 10, 2018 •

edited

Loading

eschizoid commented Sep 17, 2018 •

edited

Loading

radarwave commented Sep 26, 2018

radarwave commented Sep 26, 2018

leskin-in Sep 26, 2018

eschizoid Sep 26, 2018 •

edited

Loading

leskin-in Sep 27, 2018

eschizoid Sep 27, 2018

leskin-in left a comment

eschizoid commented Sep 27, 2018 •

edited

Loading

eschizoid Sep 27, 2018 •

edited

Loading

radarwave commented Sep 28, 2018

eschizoid commented Sep 28, 2018

radarwave commented Sep 28, 2018

HAWQ-1658. Easy Hawq and PXF initialization #1396

HAWQ-1658. Easy Hawq and PXF initialization #1396

Conversation

eschizoid commented Sep 2, 2018 • edited Loading

radarwave commented Sep 5, 2018

eschizoid commented Sep 5, 2018 • edited Loading

radarwave commented Sep 6, 2018

eschizoid commented Sep 7, 2018

ginobiliwang commented Sep 10, 2018

eschizoid commented Sep 10, 2018 • edited Loading

eschizoid commented Sep 17, 2018 • edited Loading

radarwave commented Sep 26, 2018

radarwave commented Sep 26, 2018

leskin-in Sep 26, 2018

Choose a reason for hiding this comment

eschizoid Sep 26, 2018 • edited Loading

Choose a reason for hiding this comment

leskin-in Sep 27, 2018

Choose a reason for hiding this comment

eschizoid Sep 27, 2018

Choose a reason for hiding this comment

leskin-in left a comment

Choose a reason for hiding this comment

eschizoid commented Sep 27, 2018 • edited Loading

eschizoid Sep 27, 2018 • edited Loading

Choose a reason for hiding this comment

radarwave commented Sep 28, 2018

eschizoid commented Sep 28, 2018

radarwave commented Sep 28, 2018

eschizoid commented Sep 2, 2018 •

edited

Loading

eschizoid commented Sep 5, 2018 •

edited

Loading

eschizoid commented Sep 10, 2018 •

edited

Loading

eschizoid commented Sep 17, 2018 •

edited

Loading

eschizoid Sep 26, 2018 •

edited

Loading

eschizoid commented Sep 27, 2018 •

edited

Loading

eschizoid Sep 27, 2018 •

edited

Loading